Papers with cross-modal attention module
An Effective Span-based Multimodal Named Entity Recognition with Consistent Cross-Modal Alignment (2024.lrec-main)
Copied to clipboard
| Challenge: | Existing approaches to name entity recognition rely on word-based sequence labeling and align image and text at inconsistent semantic levels. |
| Approach: | They propose a span-based method which achieves a more consistent multimodal alignment from the perspectives of information-theoretic and cross-modal interaction. |
| Outcome: | Experiments on two datasets show that SMNER outperforms the state-of-the-art methods. |
Robust and Interpretable Grounding of Spatial References with Relation Networks (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Existing models for understanding spatial references in text are vulnerable to noise in input text or state observations. |
| Approach: | They propose a text-conditioned relation network with a cross-modal attention module to capture fine-grained spatial relations between entities and a model that is robust and interpretable. |
| Outcome: | The proposed model improves performance on three tasks with a 17% improvement in predicting goal locations and a 15% improvement in robustness compared to state-of-the-art systems. |